PROJECT FA4
We are provided with a dataset of Neutritional Facts on which we are performing necessary cleaning, EDA and building various Machine learning Models. Python Libraries used for this project are-
- Pandas : For data handling, manipulation and analysis
- Matplotlib : For data visualization and analysis
- Seaborn : For Data visualization and analysis
- Numpy : For various Mathematical operations
- Sklearn.linear_model : For implementing various models
- Plotly : For online graphing tools for individuals and collaboration
- Plotly.express : For high level interaction with graphs
- scipy.stats : For scientific computations as statistics with utility functions
- sklearn : For implementing machine learning algorithms
- statsmodels.formula.api : Used for statistical models mainly works on strings and dataframes
- autoviz : Performing automatic visualization of a dataset
Initial Insights From Data
Importing necessary Libraries:
In [128]:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.linear_model import LinearRegression
import plotly
import plotly.express as px
from scipy.stats import chi2,chi2_contingency
import scipy.stats as stats
from sklearn import preprocessing
import statsmodels.formula.api as smf
from sklearn.preprocessing import MinMaxScaler
from autoviz.AutoViz_Class import AutoViz_Class
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from yellowbrick.cluster import KElbowVisualizer
from yellowbrick.cluster import SilhouetteVisualizer
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
import sklearn.metrics as metric
from sklearn.cluster import KMeans
import warnings
warnings.filterwarnings("ignore")
%matplotlib inline
Reading dataset
In [2]:
neutrients=pd.read_csv("neutrional facts.csv")
neutrients.head()
Out[2]:
Using pandas info and describe functions to gain knowledge about data type of columns, Null values, and to understand distribution of data
In [3]:
neutrients.info()
In [4]:
neutrients.describe().T
Out[4]:
In [ ]:
Following Inferences are drawn from info and describe functions
- We have 260 rows and 24 columns
- Data types of columns
- 3 are float type,
- 12 are integer type,
- 9 are object type
- Data has no Null values in it
- Mean, standard deviation and values of various quartiles are calculated in describe method
- There are three columns containing character type values i.e, Category, Item, Serving Size data but some columns containing numerical data are also of character types, so we have to convert them into numerical type i.e, integer or float
Further Exploring Category column:-
Checking unique Categories and their count
In [5]:
unique_categories=neutrients["Category"].unique()
print("Number of Unique categories =",len(unique_categories))
In [6]:
neutrients["Category"].value_counts()
Out[6]:
Plotting a count plot for checking count of vairous categories graphically
In [7]:
barchart=px.bar(data_frame=neutrients,x="Category",opacity=0.9,orientation="v",color="Category")
barchart.show()
Exploratory Data Analysis
Checking for columns containing missing values
In [8]:
missing_value_columns=[]
for i in neutrients:
for x in ["?"," ","-","Nan","v"]:
if x in list(neutrients[i]):
missing_value_columns.append(i)
break
for i in missing_value_columns:
print(i)
The output of above cell is showing columns having missing values
Replacing missing values with np.nan and then checking their counts
In [9]:
for i in missing_value_columns:
neutrients[i].replace(['?',' ','-','Nan','v'], np.nan, inplace=True)
In [10]:
neutrients.isnull().sum()
Out[10]:
The output of above cell shows count of null values in columns after replacing missing values with NAN
Converting columns to numeric type that is integer or float
In [11]:
for i in neutrients.columns:
if i not in ['Category',"Item","Serving Size"]:
neutrients[i]=pd.to_numeric(neutrients[i])
In [12]:
neutrients.info()
Now from the above info command we can see that we have 3 categorical columns and 21 numerical columns(12 int and 9 float)
Encoding Serving Size for preprocessing
In [13]:
def extract(x):
x=x.split()
oz=x[0]
return(float(oz))
neutrients["Serving Size"]=neutrients["Serving Size"].apply(extract)
In [14]:
neutrients["Serving Size"]
Out[14]:
Imputing Missing Values using Linear Regression
In [15]:
corr_df=neutrients.corr()
In [16]:
plt.figure(figsize=(15,15))
sns.heatmap(corr_df,cmap="YlGnBu",annot=True)
Out[16]:
In [17]:
values_highly_corrlated={"Calories":["Protein","Carbohydrates (% Daily Value)",'Carbohydrates','Saturated Fat', 'Total Fat','Total Fat (% Daily Value)'],"Calories from Fat":['Total Fat', 'Total Fat (% Daily Value)','Saturated Fat','Sodium (% Daily Value)','Protein'],"Saturated Fat (% Daily Value)":['Total Fat', 'Total Fat (% Daily Value)','Saturated Fat'],"Sodium":['Total Fat', 'Total Fat (% Daily Value)','Sodium (% Daily Value)'],'Vitamin A (% Daily Value)':['Trans Fat', 'Cholesterol','Carbohydrates', 'Carbohydrates (% Daily Value)'],'Iron (% Daily Value)':['Total Fat', 'Total Fat (% Daily Value)','Sodium (% Daily Value)','Dietary Fiber', 'Dietary Fiber (% Daily Value)']}
In [18]:
prediction_imputer={}
location_of_null={}
a=0
for i in values_highly_corrlated:
y=i
x=values_highly_corrlated[i]
location_of_null[y]=neutrients.loc[neutrients[y].isnull(),y].index
neutrients[y].fillna(0,inplace=True)
model_imputer=LinearRegression()
model_imputer.fit(neutrients[x],neutrients[y])
prediction_imputer[y]=model_imputer.predict(neutrients[x])
print("Accuracy for predicting",y,model_imputer.score(neutrients[x],neutrients[y]))
a=a+1
The above cell is predicting values using Linear Regression and showing accuracy of prediction
. </font>
In [19]:
print(location_of_null)
The output of above cell shows index where there were null values in columns
In [20]:
for i in location_of_null:
neutrients[i].iloc[location_of_null[i]]=prediction_imputer[i][location_of_null[i]]
The above cell is imputing values to places where there were there were nulls in columns
Inference from above code:
We have successfully imputed the missing values using Linear Regression
In [21]:
neutrients.isnull().sum()
Out[21]:
The output of the above cell shows there are no missing values
Handling outliers
In [22]:
fig = px.box(pd.melt(neutrients), x="variable", y="value", points="outliers")
fig.show()
In [23]:
for col in neutrients.drop(columns=["Item","Category","Trans Fat"]):
# Calculating values of Q1, Q3 and IQR
Q1 = neutrients.describe()[col].loc['25%']
Q3 = neutrients.describe()[col].loc['75%']
IQR = Q3 - Q1
neutrients[col].where( ~((neutrients[col] < (Q1 - 1.5 * IQR)) | (neutrients[col] > (Q3 + 1.5 * IQR))), np.NaN, inplace=True )
neutrients.fillna(neutrients[col].median(), inplace=True)
In [24]:
fig = px.box(pd.melt(neutrients), x="variable", y="value", points="outliers")
fig.show()
In Outliers handling we are omitting values which exceed our upper and lower limit
- Upper Limit=Q3+1.5*IQR
- Lower Limit=Q1-1.5*IQR
- Any value above Upper Limit and any value below Lower limit is removed by codes above for outliers handling
Statistical Tests
Statistical test between Category and Item
Since both Category and Item are categorical columns we are using 'Chi-squared' test for checking association between them
In [25]:
two_way_table=pd.crosstab(columns=neutrients["Category"],index=neutrients["Item"])
two_way_table
Out[25]:
STEP 1 Defining Null and Alternate hypothesis
- H0 -Category and Items are independent
- Ha -Category and Items are dependent
STEP 2 Selecting a significance level
- We are using .05 as significance level
STEP 3 Identification of degree of freedom
- Degree of freedom =(No of rows-1) * (No of columns-1)
- Here we have 260 rows and 9 columns so Degree of freedom=(260-1) * (9-1) = 2072
STEP 4
- Finding critical value for given significance level
In [26]:
critical_value=chi2.ppf(1-.05,2072)
critical_value
Out[26]:
STEP 5
- Computing the chi2 statistics
In [27]:
chi2_contingency(two_way_table)
Out[27]:
In output of chi2_contingency above
- First value is chi2_statistics
- Second value is P-value . Now as our chi2_statistics is less than critical value this means that our null hypothesis is failed to reject This implies that Ho is true and category and Item are independent
Since from above we can see that our pvalue is greater than .05 so we can say that null hypothesis is failed to reject
Creating a stacked bar chart for category and item
In [28]:
barchart=px.bar(data_frame=neutrients,x="Category",opacity=0.9,orientation="v",color="Item")
barchart.show()
Statistical test between total fat and Category
In [29]:
le=preprocessing.LabelEncoder()
neutrients["label_Encoded_Categories"]=le.fit_transform(neutrients["Category"])
neutrients["Total_fat"]=neutrients["Total Fat"]
In [30]:
poly_model=smf.ols(formula="label_Encoded_Categories~Total_fat",data=neutrients).fit()
In [31]:
poly_model.summary()
Out[31]:
From above output we can observe the summary of test between Category and Total Fat
Creating a bar chart between Category and Total Fat
In [32]:
barchart=px.bar(data_frame=neutrients,x="Category",opacity=0.9,orientation="v",color="Total Fat")
barchart.show()
Statistical test between Item and Protein
- Firstly we are using label encoder to encode ItemsIn [33]:
le=preprocessing.LabelEncoder()
neutrients["label_Encoded_Items"]=le.fit_transform(neutrients["Item"])
- Now we are using ols to check statistical relation between Item and Protein
In [34]:
poly_model=smf.ols(formula="label_Encoded_Items~Protein",data=neutrients).fit()
In [35]:
poly_model.summary()
Out[35]:
From above output we can observe the summary of test between Item and Protein
Statistical test between Item and Sugars
- Now we are using ols to check statistical relation between Item and Protein
In [36]:
poly_model=smf.ols(formula='label_Encoded_Items ~ Sugars',data=neutrients).fit()
In [37]:
poly_model.summary()
Out[37]:
From above output we can observe summary of test between Item and Sugars
Distribution of Calories across various menus
In [38]:
neutrients
by_category = pd.DataFrame({'Category':neutrients['Category'].value_counts().index.tolist(),
'Count':neutrients['Category'].value_counts().tolist(),
'Calories (Sum)':neutrients[['Category','Calories']].groupby('Category').sum()['Calories'],
'Calories (Mean)':neutrients[['Category','Calories']].groupby('Category').mean()['Calories']})
by_category.reset_index(drop = True)
Out[38]:
In [39]:
fig,(ax1,ax2) = plt.subplots(1,2)
fig.set_size_inches(15.5, 7.5)
ax1.set_title("Total of Items in Each Category")
ax1.pie(by_category['Count'],labels =by_category['Category'],autopct = '%1.1f%%')
ax2.set_title("Total of Calories in Each Category")
ax2.pie(by_category['Calories (Sum)'],labels =by_category['Category'],autopct = '%1.1f%%')
fig.suptitle('Calories by Category',fontsize = 20)
fig.legend(by_category['Category'],ncol=4,loc=8)
Out[39]:
- Categories having the most calories are:
- "Chicken & Fish"
- "Smoothies & Shakes"
- "Beverages"
- "Beef & Pork"
- Categories having the least calories are:
- Coffee & Tea"
- "Salads"
- "Snacks & Sides"
- "Desserts"
- "Breakfast"
Calories distribution For smoothies and shakes
In [40]:
SmoothShakes = neutrients[neutrients["Category"] == "Smoothies & Shakes"]
SmoothShakes = SmoothShakes.sort_values("Calories", ascending = False)
In [41]:
barchart=px.bar(data_frame=SmoothShakes,x="Item",y="Calories",opacity=0.9,orientation="v",color="Calories")
barchart.show()
Calories distribution For Chicken and Fish
In [42]:
Chicken_and_fish= neutrients[neutrients["Category"] == "Chicken & Fish"]
Chicken_and_fish = Chicken_and_fish.sort_values("Calories", ascending = False)
In [43]:
barchart=px.bar(data_frame=Chicken_and_fish,x="Item",y="Calories",opacity=0.9,orientation="v",color="Calories")
barchart.show()
Calories distribution For Beverages
In [44]:
Beverages= neutrients[neutrients["Category"] == "Beverages"]
Beverages = Beverages.sort_values("Calories", ascending = False)
In [45]:
barchart=px.bar(data_frame=Beverages,x="Item",y="Calories",opacity=0.9,orientation="v",color="Calories")
barchart.show()
Calories distribution For Beef & Pork
In [46]:
Beef_and_Pork= neutrients[neutrients["Category"] == "Beef & Pork"]
Beef_and_Pork =Beef_and_Pork.sort_values("Calories", ascending = False)
In [47]:
barchart=px.bar(data_frame=Beef_and_Pork,x="Item",y="Calories",opacity=0.9,orientation="v",color="Calories")
barchart.show()
Calories distribution For Coffee and Tea
In [49]:
coffee_and_tea = neutrients[neutrients["Category"] == "Coffee & Tea"]
coffee_and_tea = coffee_and_tea.sort_values("Calories", ascending = False)
In [50]:
barchart=px.bar(data_frame=coffee_and_tea,x="Item",y="Calories",opacity=0.9,orientation="v",color="Calories")
barchart.show()
Calories distribution For Salads
In [51]:
Salads = neutrients[neutrients["Category"] == "Salads"]
Salads = Salads.sort_values("Calories", ascending = False)
In [52]:
barchart=px.bar(data_frame=Salads,x="Item",y="Calories",opacity=0.9,orientation="v",color="Calories")
barchart.show()
Calories distribution For Snacks & Sides
In [53]:
Snacks_and_Sides = neutrients[neutrients["Category"] == "Snacks & Sides"]
Snacks_and_Sides = Snacks_and_Sides.sort_values("Calories", ascending = False)
In [54]:
barchart=px.bar(data_frame=Snacks_and_Sides,x="Item",y="Calories",opacity=0.9,orientation="v",color="Calories")
barchart.show()
Calories distribution For Desserts
In [55]:
Desserts = neutrients[neutrients["Category"] == "Desserts"]
Desserts = Desserts.sort_values("Calories", ascending = False)
In [56]:
barchart=px.bar(data_frame=Desserts,x="Item",y="Calories",opacity=0.9,orientation="v",color="Calories")
barchart.show()
Calories distribution For Breakfast
In [57]:
Breakfast = neutrients[neutrients["Category"] == "Breakfast"]
Breakfast = Breakfast.sort_values("Calories", ascending = False)
In [58]:
barchart=px.bar(data_frame=Breakfast,x="Item",y="Calories",opacity=0.9,orientation="v",color="Calories")
barchart.show()
Calories based on Serving Size seprately for drinks and food
- Drinks include following Categories
- Smoothies & Shakes
- Coffee & Tea
- Beverages
In [59]:
SmoothShakes = neutrients[neutrients["Category"] == "Smoothies & Shakes"]
Beverages= neutrients[neutrients["Category"] == "Beverages"]
coffe_and_tea = neutrients[neutrients["Category"] == "Coffee & Tea"]
In [60]:
drinks=pd.concat([SmoothShakes,Beverages,coffe_and_tea])
drinks
Out[60]:
In [61]:
drink_group_serving_size=drinks.groupby("Serving Size").sum()
drink_group_serving_size
Out[61]:
In [62]:
px.scatter(x=drink_group_serving_size.index,y=drink_group_serving_size["Calories"])
- Food include following Categories
- Breakfast
- Beef & Pork
- Chicken & Fish
- Salads
- Snacks & Sides
- Desserts
In [63]:
Breakfast = neutrients[neutrients["Category"] == "Breakfast"]
Beef_and_Pork= neutrients[neutrients["Category"] == "Beef & Pork"]
Chicken_and_fish= neutrients[neutrients["Category"] == "Chicken & Fish"]
Salads = neutrients[neutrients["Category"] == "Salads"]
Snacks_and_Sides = neutrients[neutrients["Category"] == "Snacks & Sides"]
Desserts = neutrients[neutrients["Category"] == "Desserts"]
In [64]:
food=pd.concat([Breakfast,Beef_and_Pork,Chicken_and_fish,Salads,Snacks_and_Sides,Desserts])
In [65]:
food_group_serving_size=food.groupby("Serving Size").sum()
In [66]:
px.scatter(x=food_group_serving_size.index,y=food_group_serving_size["Calories"])
Calories based on serving size across all menu
In [67]:
neutrients_group_serving_size=neutrients.groupby("Serving Size").sum()
In [68]:
px.scatter(x=neutrients_group_serving_size.index,y=neutrients_group_serving_size["Calories"])
Normality Test
In [69]:
#Normality test
#H0= The sample comes from a normal distribution.
#HA= The sample is not coming from a normal distribution.
Normality_df=pd.DataFrame()
dictionary={}
import scipy.stats as stats
for i in neutrients.columns.drop(["Category","Item"]):
a,b=stats.shapiro(neutrients[i])
if b<=.05:
c="The null hypothesis can be rejected"
else:
c="Null hypothesis cannot be rejected"
dictionary[i]=[a,b,c]
Normality_df=pd.DataFrame(dictionary,index=["test-statistic","p-value","Conclusion"])
Normality_df.T
Out[69]:
In [70]:
#Normality test
#H0= The sample comes from a normal distribution.
#HA= The sample is not coming from a normal distribution.
Normality_df=pd.DataFrame()
dictionary={}
import scipy.stats as stats
for i in neutrients.columns.drop(["Category","Item"]):
a,b=stats.kstest(neutrients[i],'norm')
if b<=.05:
c="The null hypothesis can be rejected"
else:
c="Null hypothesis cannot be rejected"
dictionary[i]=[a,b,c]
Normality_df=pd.DataFrame(dictionary,index=["test-statistic","p-value","Conclusion"])
Normality_df.T
Out[70]:
Normality Test uing Dist plots
In [71]:
import seaborn as sns
from scipy.stats import norm
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
df =neutrients.drop(["Category","Item","Trans Fat"],axis=1)
list_columns=df.columns
fig, ax = plt.subplots(figsize = (12, 10))
for i, col in enumerate(df.columns):
plt.subplot(5, 5, i+1)
sns.distplot(df.iloc[:,i],fit=norm, kde=False,ax=plt.gca())
#stats.probplot(df.iloc[:,i],dist="norm",plot=plt)
plt.title(list_columns[i])
#plt.axis('off')
plt.tight_layout()
Normality Test uing QQ plots
In [72]:
import seaborn as sns
from scipy.stats import norm
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
df =neutrients.drop(["Category","Item","Trans Fat"],axis=1)
list_columns=df.columns
fig, ax = plt.subplots(figsize = (15, 15))
for i, col in enumerate(df.columns):
plt.subplot(5, 5, i+1)
stats.probplot(df.iloc[:,i],dist="norm",plot=plt)
plt.title(list_columns[i])
#plt.axis('off')
plt.tight_layout()
Visualization using Autoviz
In [73]:
AV = AutoViz_Class()
In [74]:
AV.AutoViz('neutrional facts.csv')
Out[74]:
AutoViz() provides the visualization of a dataset similar to sweetviz
It provides
1) sum of columns based on their datatype
2) Pairwise plots of Variables
3) Distplots, Boxplots and Pobability plots
4) Violin plots of all Continuous variables
5) Heatmap of all Continuous variables
6) Barplots of continuous variables based on categories
Clustering Items
Pre-processing Steps
- 1) Creating copy of dataframe
- 2) Creating a Heat map
- 3) Checking For potential duplicates and dropping them
- 4) Normalization using Min-Max
Finding K for KMeans using elbow method
Predicting Clusters
- 1) Finding cluster centers
- 2) Adding clusters to dataframe
Displaying the clusters
Verifying the model using Silhouette
Verifying the model using calinski_harabasz
Step 1- Pre-processing
In [75]:
neutrient_copy=neutrients.copy()
In [76]:
corr=neutrient_copy.corr(method="spearman")
plt.figure(figsize=(20,20))
sns.heatmap(corr,cmap="YlGnBu",annot=True)
Out[76]:
Checking For potential duplicates
In [77]:
cor_matrix = corr.abs()
upper_triu = cor_matrix.where(np.triu(np.ones(cor_matrix.shape), k=1).astype(np.bool))
to_drop = [column for column in upper_triu.columns if any(upper_triu[column]>0.80) ]
print(to_drop)
Dropping columns in above cell
In [78]:
neutrient_copy.drop(to_drop,axis=1,inplace=True)
Dropping Category and Item because we had label encoded them previously
In [79]:
neutrient_copy.drop(["Category","Item"],axis=1,inplace=True)
Normalization of data using Min-Max-Scaler
In [80]:
scaled_values=neutrient_copy[['label_Encoded_Categories', 'label_Encoded_Items', 'Serving Size', 'Calories', 'Trans Fat',
'Carbohydrates', 'Dietary Fiber', 'Sugars', 'Vitamin A (% Daily Value)',
'Vitamin C (% Daily Value)']]
In [81]:
# finding the mean and std deviation of numerical columns
scaler = MinMaxScaler().fit(scaled_values)
# scaling columns to a common range
data_normalized = scaler.transform(scaled_values)# Scaled data
In [82]:
normalized_neutrients=pd.DataFrame(data_normalized,index=neutrient_copy.index,columns=scaled_values.columns)
normalized_neutrients.head()
Out[82]:
In [83]:
X=normalized_neutrients.iloc[:,2:]
X
Out[83]:
Inference from above
- In Pre-processing we have done following
- 1) Creating copy of dataframe
- 2) Creating a Heat map
- 3) Checking For potential duplicates and dropping them
- 4) Normalization using Min-Max - Since Kmean is using distance metric so we are using Normalization to increase effiency
Step 2- Elbow Method to find best K for KMeans
In [84]:
sse = []
k_rng = range(1,10)
for k in k_rng:
km = KMeans(n_clusters=k)
km.fit(X)
sse.append(km.inertia_)
plt.xlabel('K')
plt.ylabel('Sum of squared error')
plt.plot(k_rng,sse)
Out[84]:
From above graph we can observe that our elbow is at 3 and 2 but since error is less at 3 we are taking K as 3
Step 3- Predicting Clusters
In [85]:
km = KMeans(n_clusters=3)
y_predicted = km.fit_predict(X)
y_predicted
Out[85]:
cluster centers
In [86]:
km.cluster_centers_
Out[86]:
Adding Cluster column to dataframe
In [87]:
normalized_neutrients['cluster']=y_predicted
normalized_neutrients.head()
Out[87]:
Inference from above step
Since we took K as 3 we are making 3 clusters
Step 4- Clusters
In [88]:
cluster0=normalized_neutrients[normalized_neutrients["cluster"]==0].iloc[:,1:]
cluster0
Out[88]:
In [137]:
df=normalized_neutrients
df1 = df[df.cluster==0]
scalar = StandardScaler()
# fitting
scalar.fit(df1)
scaled_data = scalar.transform(df1)
# Let's say, components = 2
pca = PCA(n_components = 2)
pca.fit(scaled_data)
x_pca = pca.transform(scaled_data)
# giving a larger plot
plt.figure(figsize =(8, 6))
plt.scatter(x_pca[:, 0], x_pca[:, 1], c = df1['cluster'],cmap="viridis")
# labeling x and y axes
plt.xlabel('First Principal Component')
plt.ylabel('Second Principal Component')
Out[137]:
In [90]:
cluster1=normalized_neutrients[normalized_neutrients["cluster"]==1].iloc[:,1:]
cluster1
Out[90]:
In [134]:
df=normalized_neutrients
df2 = df[df.cluster==1]
scalar = StandardScaler()
# fitting
scalar.fit(df2)
scaled_data = scalar.transform(df2)
# Let's say, components = 2
pca = PCA(n_components = 2)
pca.fit(scaled_data)
x_pca = pca.transform(scaled_data)
# giving a larger plot
plt.figure(figsize =(8, 6))
plt.scatter(x_pca[:, 0], x_pca[:, 1], c = df2['cluster'], cmap ='inferno')
# labeling x and y axes
plt.xlabel('First Principal Component')
plt.ylabel('Second Principal Component')
Out[134]:
In [92]:
cluster2=normalized_neutrients[normalized_neutrients["cluster"]==2].iloc[:,2:]
cluster2
Out[92]:
In [130]:
df=normalized_neutrients
df3 = df[df.cluster==2]
scalar = StandardScaler()
# fitting
scalar.fit(df3)
scaled_data = scalar.transform(df3)
# Let's say, components = 2
pca = PCA(n_components = 2)
pca.fit(scaled_data)
x_pca = pca.transform(scaled_data)
# giving a larger plot
plt.figure(figsize =(8, 6))
plt.scatter(x_pca[:, 0], x_pca[:, 1], c = df3['cluster'], cmap ='plasma')
# labeling x and y axes
plt.xlabel('First Principal Component')
plt.ylabel('Second Principal Component')
Out[130]:
Displaying all clusters
In [126]:
scalar = StandardScaler()
# fitting
scalar.fit(normalized_neutrients)
scaled_data = scalar.transform(normalized_neutrients)
# Let's say, components = 2
pca = PCA(n_components = 2)
pca.fit(scaled_data)
x_pca = pca.transform(scaled_data)
# giving a larger plot
plt.figure(figsize =(8, 6))
plt.scatter(x_pca[:, 0], x_pca[:, 1], c = normalized_neutrients['cluster'], cmap ='plasma')
# labeling x and y axes
plt.xlabel('First Principal Component')
plt.ylabel('Second Principal Component')
Out[126]:
Step 5- Verifying the model using Silhouette
In [95]:
#Double check by the Silhouette score
# Instantiate the clustering model and visualizer
model = KMeans()
visualizer = KElbowVisualizer(
model, k=(2,12), metric='silhouette', timings=False
)
visualizer.fit(X) # Fit the data to the visualizer
visualizer.show()
Out[95]:
- Inference from above code
- The best value is 1 and the worst value is -1. Values near 0 indicate overlapping clusters. Negative values generally indicate that a sample has been assigned to the wrong cluster, as a different cluster is more similar.
Step 6- calinski_harabasz
In [96]:
metric.calinski_harabasz_score(X, km.labels_)
Out[96]:
In [97]:
visualizer = KElbowVisualizer(km, k=(2,12), metric='calinski_harabasz', timings=True)
visualizer.fit(X)
visualizer.show()
Out[97]:
Inference from above
- The Calinski-Harabasz index varies between 0 (worst classification) and infinity(better classification).
Predicting Calories
Selecting the predictors
- 1) Created pairplot
- 2) Creating a Heat map
- 3) Checking higly correlated columns with Calories
Splitting the data
Creating the model
- 1) Checking Coefficient and intercept
- 2) Checking Acuuracy of model
Validating Model
- 1) Validating Linear Assumption
- 2) Validating normal distribution of errors
- 3) Validating Homoscedasity principle
- 4) Calculating the VIF for each attributes
Checking Model Summary
Since Calories is a continous variable we are using a Regression model to predict it
Step 1- Deciding the Predictors with Calories as target variable
Plotting a Pair plot for checking a linear relationship with Calories
In [98]:
plt.figure(figsize=(25,25))
#sns.pairplot(neutrients)
Out[98]:
Plotting a Pair plot for checking a linear relationship between 'Serving Size', 'Calories from Fat', 'Total Fat', 'Total Fat (% Daily Value)', 'Saturated Fat','Saturated Fat (% Daily Value)' and Calories
In [99]:
sns.pairplot(x_vars=['Serving Size', 'Calories', 'Calories from Fat',
'Total Fat', 'Total Fat (% Daily Value)', 'Saturated Fat',
'Saturated Fat (% Daily Value)'],y_vars=["Calories"],data=neutrients)
Out[99]:
Plotting a Pair plot for checking a linear relationship between 'Trans Fat', 'Cholesterol', 'Cholesterol(% Daily Value)', 'Sodium', 'Sodium (% Daily Value)','Carbohydrates', 'Carbohydrates (% Daily Value)', 'Dietary Fiber', 'Dietary Fiber (% Daily Value)', 'Sugars' and Calories
In [100]:
sns.pairplot(x_vars=['Trans Fat', 'Cholesterol',
'Cholesterol (% Daily Value)', 'Sodium', 'Sodium (% Daily Value)',
'Carbohydrates', 'Carbohydrates (% Daily Value)'],y_vars="Calories",data=neutrients)
Out[100]:
Plotting a Pair plot for checking a linear relationship between 'Dietary Fiber', 'Dietary Fiber (% Daily Value)', 'Sugars', 'Protein', 'Vitamin A (% Daily Value)', 'Vitamin C (% Daily Value)', 'Calcium (% Daily Value)', and Calories
In [101]:
sns.pairplot(x_vars=['Dietary Fiber',
'Dietary Fiber (% Daily Value)', 'Sugars', 'Protein',
'Vitamin A (% Daily Value)', 'Vitamin C (% Daily Value)',
'Calcium (% Daily Value)'],y_vars="Calories",data=neutrients)
Out[101]:
Plotting a Pair plot for checking a linear relationship between 'Iron (% Daily Value)','label_Encoded_Categories', 'Total_fat', 'label_Encoded_Items', and Calories
In [102]:
sns.pairplot(x_vars=['Iron (% Daily Value)',
'label_Encoded_Categories', 'Total_fat', 'label_Encoded_Items'],y_vars="Calories",data=neutrients)
Out[102]:
Plotting a Heat map for checking a linear relationship with Calories
In [103]:
plt.figure(figsize=(20,20))
sns.heatmap(neutrients.corr(method="spearman"),cmap="YlGnBu",annot=True)
Out[103]:
By Analysing the heat map and various pairplots we reach at a conclusion that following variables hold a linear relationship with calories
- Saturated Fat
- Cholesterol
- Total Fat
- Sodium
- Trans Fat
So we are considering above variables as predictors
In [104]:
features=neutrients[["Saturated Fat","Protein","Cholesterol","Trans Fat","Total Fat"]]
target=neutrients["Calories"]
Step 2 - Splitting the Data
In [105]:
X_train,X_test,Y_train,Y_test=train_test_split(features,target,test_size=.35,random_state=100)
For Splitting we are considering 65% as train data and 35% as test data
Step 3- Creating and training a Multiple Linear Regression Model
In [106]:
model=LinearRegression()
model.fit(X_train,Y_train)
Out[106]:
Checking the Coeff and Intercept values
In [107]:
print("Coefficient :",model.coef_)
In [108]:
print("Intercept :",model.intercept_)
Checking Accuracy of model with training and testing data
In [109]:
print("Training Accuracy =",model.score(X_train,Y_train))
print("Testing Accuracy =",model.score(X_test,Y_test))
Step 4- Validating Model
Validating Linear Assumption
In [110]:
prediction=model.predict(features)
In [111]:
sns.residplot(prediction.reshape(-1),neutrients["Calories"],lowess=True)
Out[111]:
- Inference from above graph
- By Analysing the above graph we can say that predictors have a linear relationship with target as no particular pattern is observed
Validating normal distribution of errors
In [112]:
from scipy import stats
residual_error=neutrients['Calories']-prediction.reshape(-1)
plt.figure(figsize=(7,7))
stats.probplot(residual_error, dist='norm', plot=plt)
plt.title("Normal Q-Q Plot")
Out[112]:
For the above data set, the Q-Q plot of the residuals of the best fit model (shown below) indicates the adherence to normality assumption since the residual points are close to the normal line.
Validating Homoscedasity principle
In [113]:
sd=np.sqrt(np.sum(np.square(residual_error))/(residual_error.size-2))
# standardised residuals
sd_error=residual_error/sd
sq_abs_sd_err=np.sqrt(np.abs(sd_error))
plt.figure(figsize=(7,7))
sns.regplot(prediction.reshape(-1), sq_abs_sd_err,
scatter=True,
lowess=True,
line_kws={'color': 'red', 'lw': 1, 'alpha': 0.8})
plt.ylabel("Standarized residuals")
plt.xlabel("Fitted value")
Out[113]:
The linear regression model is said to abide by the homoscedasticity assumption if there is no specific pattern observed in the scale-location plot. The scale-location plot of the best fit model for the computer repairs data set is as shown below.
Calculating the VIF for each attributes
In [114]:
from statsmodels.stats.outliers_influence import variance_inflation_factor
#calculating the VIF for each attributes
vif = pd.Series([variance_inflation_factor(features.values,idx)
for idx in range(features.shape[1])],
index=features.columns)
print(vif)
From above we can say predictors are not highly correlated
Step 5- Checking Model Summary
In [115]:
import statsmodels.api as sm
In [116]:
x=sm.add_constant(features)
y=target
In [117]:
model_summ=sm.OLS(y,x).fit()
In [118]:
model_summ.summary()
Out[118]:
In [ ]: